An assessment of statistical guidance for authors at influential academic journals across scientific disciplines: Preliminary analysis
Analysis performed by: Maia Salholz-Hillel & Tom Hardwicke
Last updated: 16 August, 2021
1 Introduction
We have now completed data collection for the first phase of this project. We extracted author-facing statistical guidance provided by the top-15 journals (by Impact Factor) in each of 22 scientific disciplines (i.e., total N = 330 journals). The preregistered study protocol is available here. Below we present a preliminary analysis of the phase one data and highlight some issues to discuss as a team.
2 Methods summary
Operational definition of statistical guidance: any advice or instruction related to the appropriate selection, implementation, reporting, or interpretation of statistical analyses.
We recorded whether each journal:
- Provided any statistical guidance at all
- Had a dedicated statistical guidance section
- Referred to statistical guidance in external sources (e.g., reporting guidelines or articles)
If any statistical guidance was provided by a journal:
- We coded (YES/NO) whether it mentioned each of 20 statistical topics (e.g., “p-values”), and extracted the verbatim text.
Note: When journals referred to guidance in external sources, we have recorded the names of the sources, but we have not currently extracted guidance from those sources. We return to this point in the issues to address section (Section 4) at the end of this document.
All data extraction was performed by a first coder (DS, MM, TB, MSH) and a second coder (TEH) with any coding differences resolved through discussion.
Note: We found that some publishers provided statistical guidance that was shared across their portfolio of journals (specifically, 31 Nature journals, 12 Cell journals, and 2 Frontiers journals). Shared publisher guidance was coded by two team members (as above), but is represented in our data and analysis multiple times - once for each individual journal it applies to. For example, the Cell journals guidance is inherited by 12 journals so appears 12 times in the dataset. Unless stated otherwise, publisher guidance is treated as journal-specific guidance (i.e., not external guidance) in the analyses presented below.
3 Results
Figure 1 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that (a) offered any statistical guidance; (b) had a dedicated statistical guidance section. For tabular data, see Appendix 5.
You can see that around half (48%) of journals offered any statistical guidance. Note that this includes 32 journals which only referred to statistical guidance in external sources (reporting guidelines or academic papers). Just over quarter of journals had a dedicated statistical guidance section of their author instructions (28%). In two fields (Computer Science and Maths), no journals offered any statistical guidance. Journals in health-related fields were more likely to offer statistical guidance and have dedicated statistical guidance sections. Notably, 100% of the journals in clinical medicine offered some statistical guidance.
Figure 1: Percentage of journals offering statistical guidance by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.
Figure 2 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that mentioned each of the twenty pre-specified statistical topics. For tabular data, see the Appendix (Section 5). Please see Section 4 (issues) at the end of this document for an important note regarding the topic “prespecification of analyses”.
Figure 2: Percentage of journals offering guidance on twenty statistical topics by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). For presentational purposes, disciplines offering no guidance on individual topics are shown, but not labelled. Two scientific disciplines in which no journals offered any statistical guidance at all are not shown. Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.
You can see that some topics (e.g., confidence intervals, p-values) are much more likely to be mentioned than other topics (e.g., handling outliers, categorisation of continuus data). Journals in health-related fields, particularly clinical medicine, are often more likely to mention these topics than journals in other fields.
For the 128 journals that offered some internal guidance (i.e., excluding those that offered no guidance at all and those that only referred to external sources), the histogram in Figure 3 illustrates that the maximum number of topics mentioned by an individual journal was 15. There were 9 journals that did not mention any of our prespecified topics (but provided statistical guidance on other topics). The median topics mentioned was 6.
Figure 3: Histogram showing how many of the twenty statistical topics were mentioned by each journal
In the grid below (Figure 4), you can see which of the twenty topics were mentioned by each journal. Journals that shared publisher guidance are represented by one row (per publisher). Journals that did not offer any statistical guidance at all, or journals that referred only to guidance in external sources, are not shown.
Figure 4: Grid diagram showing whether each journal provdied guidance on each of twenty prespecified statistical topics
Look at external guidance
| external_guidance | n_journals | guidance_type |
|---|---|---|
| consort | 95 | reporting guideline |
| arrive | 80 | reporting guideline |
| prisma | 47 | reporting guideline |
| remark | 40 | reporting guideline |
| strobe | 39 | reporting guideline |
| stard | 36 | reporting guideline |
| equator | 25 | reporting guideline |
| icmje | 21 | reporting guideline |
| cheers | 18 | reporting guideline |
| strega | 16 | reporting guideline |
| nih principles and guidelines for reporting preclinical research | 14 | reporting guideline |
| moose | 12 | reporting guideline |
| care | 11 | reporting guideline |
| miame | 11 | reporting guideline |
| coreq | 9 | reporting guideline |
| spirit | 9 | reporting guideline |
| tripod | 9 | reporting guideline |
| brisq | 8 | reporting guideline |
| squire | 8 | reporting guideline |
| gather | 7 | reporting guideline |
| sampl | 6 | reporting guideline |
| srqr | 6 | reporting guideline |
| agree | 5 | reporting guideline |
| ispor | 5 | reporting guideline |
| mibbi | 5 | reporting guideline |
| trend | 5 | reporting guideline |
| cope | 4 | reporting guideline |
| prisma-p | 4 | reporting guideline |
| aapor | 3 | reporting guideline |
| cummings & rivara 2003 (10.1001/archpedi.157.4.321) | 3 | paper |
| orion | 3 | reporting guideline |
| spirit-ai | 3 | reporting guideline |
| apa jars | 2 | reporting guideline |
| cumming et al. 2007 (10.1083/jcb.200611141) | 2 | paper |
| grips | 2 | reporting guideline |
| nlm research reporting guidelines and initiatives (https://www.nlm.nih.gov/services/research_report_guide.html) | 2 | reporting guideline |
| olsen 2003 (10.1128/iai.71.12.6689-6692.2003) | 2 | paper |
| olsen 2014 (10.1128/iai.00811-13) | 2 | paper |
| richardson & overbaugh 2005 (10.1128/jvi.79.2.669-676.2005) | 2 | paper |
| spqr | 2 | reporting guideline |
| altman et al. 1983 (10.1136/bmj.286.6376.1489) | 1 | paper |
| apa manual | 1 | publisher |
| asa (https://www.amstat.org/asa/your-career/ethical-guidelines-for-statistical-practice.aspx) | 1 | publisher |
| biosharing information resource | 1 | reporting guideline |
| boushey et al. 2006 (10.1016/j.jada.2005.11.007) | 1 | paper |
| boushey et al. 2008 (10.1016/j.jada.2008.01.002) | 1 | paper |
| bruemmer et al. 2009 (10.1016/j.jada.2009.07.011) | 1 | paper |
| consort abstract extension | 1 | reporting guideline |
| consort extended guidelines | 1 | reporting guideline |
| consort pro extension | 1 | reporting guideline |
| entreq | 1 | reporting guideline |
| fda guidelines (https://perma.cc/w2ey-msta) | 1 | reporting guideline |
| fiar | 1 | reporting guideline |
| gleason et al. 2010 (10.1016/j.jada.2009.11.022) | 1 | paper |
| gleason et al. 2015 (10.1016/j.jand.2015.03.011) | 1 | paper |
| gnosis | 1 | reporting guideline |
| harris & raynor 2017 (10.1016/j.jand.2017.03.017) | 1 | paper |
| harris et al. 2008 (10.1016/j.jada.2008.06.426) | 1 | paper |
| harris et al. 2009 (10.1016/j.jada.2008.10.018) | 1 | paper |
| harris et al. 2012 (10.1016/j.jada.2011.09.037) | 1 | paper |
| hewitt 2012 (10.1007/s10519-011-9504-z) | 1 | paper |
| hollingshead 2008 (10.1093/jnci/djn351) | 1 | paper |
| http://www.biostathandbook.com/ | 1 | paper |
| http://www.utdallas.edu/~serfling/3332/biology_statistics_made_simple_using_excel.pdf | 1 | paper |
| https://www.ncbi.nlm.nih.gov/books/nbk153593/ | 1 | paper |
| human genome epidemiology network (hugenet) guidelines | 1 | reporting guideline |
| iucr | 1 | reporting guideline |
| kempen 2011 (10.1016/j.ajo.2010.08.047) | 1 | paper |
| miqe | 1 | reporting guideline |
| motulsky 2014 (10.1124/jpet.114.219170) | 1 | paper |
| poldrack et al. 2008 (10.1016/j.neuroimage.2007.11.048) | 1 | paper |
| qhes | 1 | reporting guideline |
| record | 1 | reporting guideline |
| sheean 2011 (10.1016/j.jada.2010.10.010) | 1 | paper |
| simon et al. 2009 (10.1093/jnci/djp335) | 1 | paper |
| strenda | 1 | reporting guideline |
| strobe-me | 1 | reporting guideline |
| strond | 1 | reporting guideline |
| sullivan et al. 2016 (10.1161/jaha.116.004142) | 1 | paper |
| table 3 in dupuy & simon 2007 (10.1093/jnci/djk018) | 1 | paper |
| tidier | 1 | reporting guideline |
| zoellner & harris 2017 (10.1016/j.jand.2017.01.018) | 1 | paper |
| zoellner et al. 2015 (10.1016/j.jand.2015.03.010) | 1 | paper |
External guidance types
| guidance_type | n |
|---|---|
| paper | 29 |
| publisher | 2 |
| reporting guideline | 52 |
4 Issues to address
Our coding for the topic “prespecification of analyses” was not consistent across coders. Specifically, some coders considered guidance/instructions related to registration of clinical trials to be covered by this topic, whilst others did not. I believe the justification of those who did not was that clinical trials registration does not necessarily involve prespecification of a statistical analysis plan. That seems a reasonable justification to me, but I’d be interested to hear thoughts from the team. If we decide that advice about clinical trials registration alone does not fall into the remit of this topic, then it is relatively straightforward to examine the verbatim text we extracted and re-code the relevant cases. However, if we decide that advice about clinical trials registration does fall under the remit of this topic, then we will need to re-visit the journal instructions to authors and search for such guidance (as this was not done consistently previously).
We have recorded whether journals refer to statistical guidance from external sources, such as reporting guidelines or academic papers. However, we have not examined those external sources, and we need to decide whether to do so. If we perform extraction and coding for all 83 external sources we have identified, this is clearly a lot of extra work. We also perhaps only really interested in the most salient statistical guidance offered by journals. On the other hand, if we do not extract information from external sources, then perhaps we are missing an important aspect of statistical guidance i.e., that journals may not provide their own guidance on certain topics because they are aptly covered by external sources. An additional complication is that in some cases, the boundary of internal vs external guidance is blurred when the ‘external’ source is an academic article that appears to have been written by members of the journal’s editorial team.
5 Appendix
Tabular data (represented in Figures 1 and 2). There is a scroll bar at the bottom of the table to see all of the scientific fields.
| Characteristic | Overall, N = 330 | AGRICULTURAL SCIENCES, N = 15 | BIOLOGY & BIOCHEMISTRY, N = 15 | CHEMISTRY, N = 15 | CLINICAL MEDICINE, N = 15 | COMPUTER SCIENCE, N = 15 | ECONOMICS & BUSINESS, N = 15 | ENGINEERING, N = 15 | ENVIRONMENT_ECOLOGY, N = 15 | GEOSCIENCES, N = 15 | IMMUNOLOGY, N = 15 | MATERIALS SCIENCE, N = 15 | MATHEMATICS, N = 15 | MICROBIOLOGY, N = 15 | MOLECULAR BIOLOGY & GENETICS, N = 15 | Multidisciplinary, N = 15 | NEUROSCIENCE & BEHAVIOR, N = 15 | PHARMACOLOGY & TOXICOLOGY, N = 15 | PHYSICS, N = 15 | PLANT & ANIMAL SCIENCE, N = 15 | PSYCHIATRY_PSYCHOLOGY, N = 15 | SOCIAL SCIENCES, GENERAL, N = 15 | SPACE SCIENCE, N = 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| has_guidance | 160 (48%) | 5 (33%) | 13 (87%) | 4 (27%) | 15 (100%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 8 (53%) | 1 (6.7%) | 14 (93%) | 8 (53%) | 0 (0%) | 9 (60%) | 12 (80%) | 9 (60%) | 11 (73%) | 11 (73%) | 5 (33%) | 11 (73%) | 13 (87%) | 6 (40%) | 2 (13%) |
| has_external_guidance | 137 (42%) | 4 (27%) | 12 (80%) | 2 (13%) | 15 (100%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 13 (87%) | 7 (47%) | 0 (0%) | 8 (53%) | 12 (80%) | 8 (53%) | 11 (73%) | 11 (73%) | 5 (33%) | 5 (33%) | 13 (87%) | 3 (20%) | 1 (6.7%) |
| has_internal_guidance | 128 (39%) | 5 (33%) | 12 (80%) | 4 (27%) | 14 (93%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 7 (47%) | 1 (6.7%) | 12 (80%) | 4 (27%) | 0 (0%) | 7 (47%) | 12 (80%) | 9 (60%) | 7 (47%) | 5 (33%) | 4 (27%) | 8 (53%) | 8 (53%) | 5 (33%) | 1 (6.7%) |
| has_internal_guidance_section | 93 (28%) | 2 (13%) | 11 (73%) | 2 (13%) | 9 (60%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 8 (53%) | 3 (20%) | 0 (0%) | 5 (33%) | 11 (73%) | 9 (60%) | 6 (40%) | 2 (13%) | 4 (27%) | 7 (47%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) |
| has_p_value | 88 (27%) | 2 (13%) | 9 (60%) | 1 (6.7%) | 13 (87%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 10 (67%) | 3 (20%) | 0 (0%) | 3 (20%) | 5 (33%) | 8 (53%) | 6 (40%) | 2 (13%) | 4 (27%) | 4 (27%) | 7 (47%) | 2 (13%) | 0 (0%) |
| has_significance | 52 (16%) | 2 (13%) | 4 (27%) | 1 (6.7%) | 7 (47%) | 0 (0%) | 2 (13%) | 0 (0%) | 2 (13%) | 0 (0%) | 4 (27%) | 0 (0%) | 0 (0%) | 2 (13%) | 7 (47%) | 5 (33%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 2 (13%) | 6 (40%) | 3 (20%) | 0 (0%) |
| has_null_hypo | 33 (10%) | 0 (0%) | 6 (40%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 3 (20%) | 2 (13%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_sample_size | 72 (22%) | 3 (20%) | 10 (67%) | 3 (20%) | 7 (47%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 3 (20%) | 0 (0%) | 4 (27%) | 11 (73%) | 5 (33%) | 5 (33%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 2 (13%) | 0 (0%) |
| has_conf_int | 90 (27%) | 1 (6.7%) | 12 (80%) | 2 (13%) | 13 (87%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 9 (60%) | 3 (20%) | 0 (0%) | 5 (33%) | 11 (73%) | 6 (40%) | 7 (47%) | 1 (6.7%) | 4 (27%) | 2 (13%) | 5 (33%) | 1 (6.7%) | 0 (0%) |
| has_effect_size | 61 (18%) | 0 (0%) | 7 (47%) | 1 (6.7%) | 10 (67%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 1 (6.7%) | 7 (47%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 4 (27%) | 5 (33%) | 0 (0%) | 4 (27%) | 2 (13%) | 5 (33%) | 1 (6.7%) | 0 (0%) |
| has_multi_compare | 60 (18%) | 0 (0%) | 9 (60%) | 1 (6.7%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 3 (20%) | 0 (0%) | 4 (27%) | 5 (33%) | 7 (47%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 5 (33%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_subgroup | 30 (9.1%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 9 (60%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 7 (47%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_baseline_covar | 9 (2.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 5 (33%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_non_param | 10 (3.0%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 3 (20%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_sensitivity | 11 (3.3%) | 0 (0%) | 0 (0%) | 0 (0%) | 5 (33%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_model_assume | 61 (18%) | 1 (6.7%) | 9 (60%) | 2 (13%) | 4 (27%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 4 (27%) | 11 (73%) | 7 (47%) | 5 (33%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) |
| has_exclusion | 64 (19%) | 1 (6.7%) | 10 (67%) | 2 (13%) | 5 (33%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 3 (20%) | 10 (67%) | 6 (40%) | 4 (27%) | 2 (13%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 0 (0%) |
| has_outliers | 7 (2.1%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 3 (20%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_missing | 18 (5.5%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 6 (40%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 3 (20%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 0 (0%) |
| has_one_sided | 51 (15%) | 0 (0%) | 8 (53%) | 1 (6.7%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 7 (47%) | 4 (27%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 0 (0%) |
| has_bayes | 39 (12%) | 0 (0%) | 7 (47%) | 1 (6.7%) | 2 (13%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 6 (40%) | 3 (20%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_secondary | 25 (7.6%) | 2 (13%) | 1 (6.7%) | 0 (0%) | 10 (67%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 4 (27%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 3 (20%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_prespecify | 81 (25%) | 2 (13%) | 8 (53%) | 1 (6.7%) | 14 (93%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 7 (47%) | 3 (20%) | 0 (0%) | 5 (33%) | 6 (40%) | 8 (53%) | 5 (33%) | 2 (13%) | 4 (27%) | 3 (20%) | 8 (53%) | 1 (6.7%) | 0 (0%) |
| has_cat_continuous | 2 (0.6%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) |
| has_publisher_guidance | 46 (14%) | 0 (0%) | 8 (53%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 4 (27%) | 10 (67%) | 3 (20%) | 3 (20%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) |